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InCroducGion 

Project KIDS (Kindling Individual Development Syscaflis)j under the 
direction of Dv* Ruth Turner ^ is a Dallas rndepetidint School District 
model pifogram for handicappad infants i toddlers , preschool children and 
Chair families • Beginning in the .197 5-76 year, The Bareaa of Education 
for the H^dicapped (BEH) provided chree'-year support for the project under 
the Handicapped Children's Early Intervention program? and the Dallas ISD 
assumed support for the project beginning the 1978'-79 school year. However , 
BEHi provided an additional three-yeat grant to fund Project KIDS OUTREACH, 
which Is an esctenslon of the KIDS modtl into additional Dallas ISD early 
childhood classrooms and into surrounding suburban school dlstriccs* 

Dr, Turne-r designed the KIDS model to serva developaentally delayed 
and physicall/ handicapped preschool children age birth Co bLk years. 
Services are delivered through home-'based instruction, ceii ter^baaed in^ 
fant iCimulatton classes j and sehool--tased early childhood class units. 
The primary instructional vehicle is the Mini A^ctivity Plan (MAP) which 
details the iadlvidualized educational plan for each child in reference 
to observed performance in the KIDS Inventory of Development, Each develop-* 
mental item in the Inventory cross references a variety of curricula and 
activities to promote the developmental skill or behavlot tasted by that 
particuLar item* Hencej projecC staff can plan instructional intervention 
in direct response to assessed perf orx^ance levels* 

Another Icey feature of the KIDS tnodel is the Integration of the child's 
parents and family Into a cooperative instructional role with project staff* 
In the home'^based cotnponentj the parent is seen as the pt-lmary instructional 
agent with more instruction being assumed by staff as the child progresses 



toward school-based instruction. A. competency based parent itwolvement 
program has been developed to provide iiidit.'idualized trainini £or parents 
(Turner, 1978), This program provides a continuum of tratrtini activities 
which parents selact in ternis of selfi-repotted competency leytls (Carter, 
1978; Macy, 1978), 

Project KIDS also receives supplementary appraisal sM\?lces froffl the 
University Affiliated Center of the University cE Terns Health Science Cen- 
ter and faculty from the Special Education Program at the Untvarsity of TeKas 
at Dallas also provides consultative assistance to the project* 

The purpose of the current study was to secure comparactve data for 
evaluating the impact of project intervfintion on developTnental progress of 
children. As one might expect ^ it is virtually impossible to identify 
a random control group for the population served in Project KIDS, and identi-- 
flcation of even a comparison group is almost iuipossible , or certainly 
unrealistic* In the absence of a control gtoup or a comparlscin group of 
children^ one must find an alternative point of reference for cornparison. 
One approach to obtaining a comparative point of refarence In early intar-= 
vention has been to calculate an expectancy score based on precast perfor-" 
manca* The basic procedure is to compute a developmental rate as the ratio 
of developmental age to chronological age at the time of pt&test. Stmeonsson 
and Wiegerink (1975) bul^t on this concept in suggestini an Index of effi- 
ciency for comparing heterogeneous children within a project or across pro^ 
jects, 

toother approach to obtaining a comparison point of reference is to 
adopt a time lag design and form a comparison group or groups within the 
project as a function of age at time of pre-^post measurement* The basic 



primise of this time lag approach Is chat the pretest scores of one group 
can serve as the cQmparlsoti for posttest scores of another group. For a 
discussioti of time lag designj see Goulet (1975), 

Another strategy frequently used In Special education research is the 
matched pairs design* The popularity of this design is largely due to the 
difficulty in securing adequate control groups iri special education popula-^ 
tions , 

While the above research strategies have merit and potential applica-' 
tiotij none is the design of choice due to probieitis encountered in special 
subject populations such as the population servad in Project KIDS. The com-- 
parison of developmental rates (developTiiental agt r chronological age) at 
ptecest and posttest is one of the most popular research strategies in use 
with the early childhood handicapped populatioti. However, two reserva^ 
tions come to mind when approach is used ^Ith very young children. 
The first centers about Che stability In the rate of maturation during 
the early months and years of life. If in fact one can assume that the matura 
tion rate is linear throughout the first 60 irioriths of life, then one might 
place greater confidence in comparison of pre^poac developmental rates , but 
the aasumption of linearity in maturational rate appears tenuous at best. 

Comparison of pre-past developmental irates certainly has the potential 
to demonstrate significant pre--post gains ^ but an alternative to project 
intervention as an explanation of these gains couLd easily be an artifact of 
possible curvilinear ity In maturaclonal rate. For example, if an appreci- 
able number of children in the ptroject were pretested at an age just prior 
to a normal increase in taaturac tonal rate in the dependent variable^ the 
observed pre-*post gains could be largely due to just normal maturation and 
not necessarily to project intervention. This ptienomenon could also work to 
decrease or wash out real gains of children due to project intervention* 



A secotid reservation about the developmental rate strategy has to do 
with the ifflpact of measuremant error on the ratio of developinental age to 
chronological age, and especially so when ages involve small numbers. For 
example^ the standard error of measurement for the Bayley mental scale is 
6.7 tm acore units for a 12-month old child (Bayley, 1969). If such a 
child were to have a pretest developmeiical age of 7 morithSj the developmental 
rate would be ,58, but if the precest raw score were orily one standard error 
higher^ the child's developmental age would be 9 months. This would change 
the pretest developmental rate from .38 to .75, simply as a result of 
measurement error. Such variation could influence the interpretation of pre- 
post developmental rate comparisons. 

Another approach to obtaining a comparison point of reference would be 
to adopt a kind of time lag design and form a comparlaon group or groups 
wlthlri the project as a function of age at time of pre^post measurenient . 
The foLlowing outlines a schema of a possible time lag approach in Project 
KIDS. 

Age in months 

Treatment 

Population 12 24 36 48 

Home-based pre — - — ^ post 

Center'-based pre — — ^ post 

School-based P^e ^ post 

In the time lag strategy the pretest scores of one group serve as the 
comparison point for the posttest scores of another group* In Project KlDSs 
the comparison of pretest scores of center^based chlldreTi to posttesc scores 
of home-based children could be used to test the effectiveness of home-based 



inatruction- Similarly, one could compare preteit school^^bassd to postteac 
Cfenter-based Co test effectives of centers-based instructions but there would 
be no comparison for school-'basad Instxact ion* One obvious linti cation to the 
tlnie lag approach is the inabiliciy to assign randomly children Co pre and 
post comparison groups, thereby reducing experimental control. Another 
Haitation la that the time lag stirategy requires a sufficient number of 
children in each coniparlson group to yield suitable sample siges in the 
lagged pre^post coniparisons , In Project KIDS children enroll on a refarral 
basis which reduces the probability of achieving suitable sample sizes for 
the neceasaTy comparisons. 

The oatched pairs designs a third popular strategy ^ can provide a rea- 
sonably defensible designs if children can be closely matched on. all critical 
variables, Macy and Career (1975) have reported success in matching severely 
handicapped school age children 3 hut good success in matching requires a 
large populacion from which to draw, En Project KIDS Che popuiation of 
children la quite limited in sizes and the heterogeneity among children and 
families would make the matching process most difficult, or perhaps not even 
possible. 

In llghc of the limitations of the research designs usually applied in 
special education research ^ it was thought a more innovative strategy was 
needed* The concept of a theoretical coricrol strategy was consequently 
developed for the product evaluation of Project KIDS- The theoretical con"- 
trol strategy reqiiires a panel of experts to project the test score per- 
fomiance of subjects given the aisuwption of no eKperlmencal intervention* 
In other words, e^cperts are asked to score a given test as they think the 
subject would have scored at some poltit In tirne given no intervention. 



rhe cheoretlcal eontrol score is then the average score (or a desired 
ceatral tandency measure) of the scores projected by the expert panel* 

Panels of eKperts are cotranonly used in instrument validation studies » 
but chare has been no or very limited use of experts In developing theoret- 
ical controLs. Curtis and Donlon (1972) have studied agreement among 
perts in rating video cape recordings of multiply handicapped chlldrin. 
Stricklin (1974) used an expert panel to develop psychiatric profiles of 
adulcs in a clinical setting. Reported research suggested that the theoret- 
ical control strategy as conceived in the Project KIDS evaluation rnight be 
feasible, but a review of literature revealed no applications of an expert 
panel in obtaining projected performance in the context of a control coin- 
parlson. 

The theoretical control approach seemed to be especially applicable to 
Pro j act KIDS in chat the project addresses developmental delay, an area 
with sufficient background research and clinical experience to make projec- 
tions possible* The procedure was to select professionala who were recognised 
and accepted as experts In child development. These experts were independenc 
of Project KIDS and possessed substantial credentials and experience in child 
development. Each expert independently reviewed the complete assesstnent 
records of each child and then formulated an assessment profile of the child 
proj acted forward twelve months from date of pretest assessment. Experts 
based their projections on the assumption of no Intarvention, 

The cuirrent application of the theoretical control group strategy in 
Project KIDS involved a sample of 17 children (aged 18 months or less at 
pretest) and an expert panel of four nationally recognized leaders in child 
development. Panel members used the Bayley Scales of Infant Development 
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and the KIDS Inventory of Development (a developTnantal checklist) to record 
their 12«-raonth projections* In effects experts scored Che Bayley and KIDS 
Inventory as though they were actually administering a posttest Co the 
child. 

The study attempted to answtr the following research quastions^ 

1* Whac was the psychometric status of scores projected by eKperts? 

lil What waa the reliability of projected scoras? 

Is 2 What was the validity of projected scores? 

1.3 Did individual eKperts exhibit any conslstetit deviation in 
projecting scores? 

1.4 Was the variability in projected scores related to children's 
precest perf orroance , related subtest performance, age^ num= 
ber of items projeccedj handicap ^ or extent of available infor 
mat ion? 

2* Was children's devalopmental progress , if any^ due to project inter 
vencion? 

2*1 Was there significant improvement in actual pre-post scores 
of sampled childrtn? 

2,2 t^Jhat were the characteristics of theoretical control 
scores? 

2*3 How did theoretical control scores compare to actual post- 
test scores? 

Procedures 

The following describes instrumentation, sample, and data collection 
procedures used in the study. 

Ins t rumenjat ipn^ 

Expert panel members used the Bayley Scales of Infant Development 
and the KIDS Inventory of Developinent to score their projections. Other 
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"inatrumentatcioii" included the child case study data given Co expercs upon 
whieh to base their pro j actions . 

Bayley Scales of Infant DevelopTnent . The Bayley is one of the best 
kno^ InstturQents for measuring developniental status in the first two and 
one-half years of life. The Instrument yields a Mental Development Index, 
a Peychomotor Development Index, and a Behavior Record (Bayley, 1969) » The 
standardlEatiOTi saopla Included 1^262 children and was representative of the 
United States population (1960 census) In terms of urban-rural residence ^ 
whlte--nonwhice race, occupatton and education of the head of the household, 
and geographic region (of couratj the scandardiaation sample included only 
normal children). Split-half reliabilicy coefficients reported for the mental 
scale range from .81 Co .93, and coefficients for the motor scale range from 
* 68 to ,92* The mental scale contains 163 icams and the motor scale has 81 
items, 

li 

KIDS In ventory of Devel opment, The KIDS Inventory was developed prunarily 
under the direction of Dr. Nanci Bray, a University of TeKas at Dallas con- 
sultant to Projecc KIDS, and Dr , Ruth Turner, Project KIDS Director. Project 
KIDS staff also provided valuable contribution and effort to developing the 
Inventory. The Inventory is still in a developmental status and is undergoing 
further refinement and modification. The basic format of the Inventory is a 
checklist of developinental behaviors sequenced according to chronological age 
0 to 72 months in four areas of development: gross motor (77 items), fine 
motor (70 items) , language/cognitive (112 items), and self-help (64 items). 
Scoring of the inventory is pass--fail for each behavior tested. Preliminary 
study shows that the criterion validity of Che KIDS Inventory ^Ith the Bayley 
mental is .75 and ,88 with the Bayley motor CCarter, 1978)* 
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child Case Study Data. An iKtensive case study profile ms copipiled 
for each child in the study* This profile included results from eduGa tioxial 
psychQlogical J sociological » and medical assessiaents . Pretest record fortns 
from the Baylay mental and motot scales were also included in the profiles. 
k small pilot theoretical control study Involving a paneL of levea pro- 
fessionals and five children determined that the above informacioa was rele- 
vanc and useful in preparing fiKparts for the task of projecting performance* 
Since the eKtent of assessment and evaluation completed on each child varied 
the amount of Information contained in each child' s profile also varied* 
The number of pages in a profile ranged from 40 to 100? and the typical 
number of pages was 60. 



The sample for the study iticluded 17 childrea and Coiir eKpe^t panel 
members. The four panel members were nationally recogntE^d aKpetts in early 
childhood education for the handicapped- They vers: 



Dr. Bet eye Caldwell 

Center for Child Development and EducaCion 
University of A^rkansas at Little Roclc 
Little Rocks Arkansas 

Dr* Ernest GoCts 
School of Human Development 
University of TeKas at Dallas 
Richardson , Texas 

Dr. Maynard Reynolds 

Departmerit of psychoeducational Studies 
University of Minnesota 
Minneapolis , Minnesota 

Dri Francis Walker 

Office of Child Developttient 

California State Departittent of Education 

Sacramento , California 



Sample 
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Th^ sample of cbildre^ IsicLuded 17 child^eim (^even maLe snd ten £e[a.^le) 
IS GOntti mnd yauager ac tinie off^BayLfiy pretext (i*s.p entry intfl Erojec^ 
KXDS)* The aTE^age age at pretsit was Ll»68 fliorxthSi and tlii jtattdard di^la-^ 
tioti wa^ 5 ,44 laontl^s, Tha average pretest Ba:yl©y rfancaL aad l^otor ra^ stfores 
warn 66 -29 and 25.35; ttie ^taEid&rd dam^tlons y^mm 33-38 atxd 34, 64^ re- 
spetclv^ly, Ihe average mental dev^eLopnaental a^e ^^as 5,4 oon^hs 5 and th^ 
a^iraga mocor developinimal age was5.6 norths. Since the average chrron^D- 
logical agm wm 11-68 tontfc^i, ttxe avfeTage tfstitaL mstucatiors-al raCe m& 
pement (5*4 4 11,68) $ 3-nd the ^veirage tttotor wtiwas 43 p&rcent (5.6 * 
1L.5S), Nosedives from th^ KIDS Iriveiitoty we*i av^llabLa for this iatfipl^> 
SLnee ttie Invcricory v^s undergoLng initial de^eL op^ent: a t chim tiin&. 

Th^re & wide r&ng^ of t^anilc^pp ing conditions a-vldent Ici the sarSple, 
Th-i folici^ing l±sts a belief description of the aonditiona for each child 3 



1 

Ghald Mn41cg£P^n^J2S45iiSSS 

3. cfiri^Tno^omal an^ttiaiys mino^ p hysieal abaoroialitlts 

w-itlx p^ssibLe txea^iag protolea] 

2 language and p&rs^nai/^ocdaL devnj^opfleatal delay 

3 sev&re recardatloti w^ltti vision and hearing p%:oblem;i 
6 hydrocephalic, CNS damage^ trainable le^^aj 

5 B^i^Mtm disord&rj left hetfiipar©sls 

6 Cf ? tilLd Spastic paraplegia 

7 i^faTitile tn^^clonus ^ pr^ofc3uri4 brain damage 

S m^ltipie coaser^lt^l an^iaJie^ of linlciiow^ etloLoiy 

9 Dcwm's snydr^ome (S^) 
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CNS damage, hyj)Otonl,a 


U 


micro cephal ic 


la 


lamguaie and mator dal^y 


13 


CNS daiage, physLcal, ainoni4ll«s 


14 


CP a right g-pAstic haulf ar-lsLa 


15 


Down's snydsomfi C^^) 


16 


Down's snyd^oma ClIR) ) faHur-fl CO Chcrivt 


17 


devBLopfflsntsl deLay of ualno^ atftol-osy 



CNS ^ CeHtraL nervcus systetni CP - cer-^br^l palla^j ^ ^ mental 

Inspect ictiu o£ thi above Llsc of cotidltioTis la^lcsCes at least superf icially ^ 
th^ extent o£ bsndicap involvement Iti sampled chiiarem, The vast majority 
of the saOTl^ Deviously tncludad ctiiLdreTi -^itm severe handicapping conditioni, 
How\7etp the e^«tit of aevarity in the s^niple Likai^r greatar than that 
in the Coc^l Project KIPS papulations stmce linitiaL raf&rtala to the project 
tecided Co Mve ciore severe invoXveffletit, 

EKpiftpariel meinbers rsceiv^ed detailed ddr&cctons f oir scoring their 
prQjectiQff.s and aXl necessary materials. Ins^mettons directed eKperts to 
project wt%at tlxey thought ttie child's paTf^unance on t;h^ Bayley and KIDS 
In^eatoty voald ba 12 months from cha Cls^e of precestl.n^t Experts were to 
assuDS no apeclal inticvamions inCo the cHlld afld famlLy life (aside from 
any aeceapary life support for cha child), 4 coCTpLiCa&it of directions to 
experts ap^esrs in. the Appefidix* 
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Results from a preliminary pilot mtuiy t^lth local experts showed that 
one hour was the typical time required to ^aview a child's casa study and 
complete projected test score perfDrma^ce, However ^ expert panel members 
were not asked to record their tlnie in conipLfiting the score projections.^ 
Two of the four eKpefts ware prompt in returning their completed projections , 
and the other two experts delayed in completing their proj ecttoni * 

Results 

The results of the study are reported by individugil resiarch ques- 
cions. 

1,1 What was the realiability of projected scores? 

Data analysis for this question utilised analysis of variance co 
estimate reliability of expert proj eccioris . The procedure foIlQwed was 
Chat as described by Winar (197, pp, 283 £fO. Application qC this proce^ 
dure yielded what is known as an intMcUss correlatioti coefflclint^ which 
provided an estiinata of reliability for any one eKpert . The Spearnian-Browri 
prediction formula was used to obtain che Teliabillty estltnace £or the 
average projected score from all fou^ eKperts, 

Thm general computational procedura was to calcuiate the unbiased esti- 
mate of the ratio of true score variance to error score varimce. This 
ratio, tamed theta, was then used the following equation to compute ra- 
liability coefficients: 

r ^ .-^-r^ , where k is the amtar of 
experts, an 0 is the ratio of true sco« variance to error acore mriance. 
Sunmary tables for ANOVAs and detailed reliaollity calculations are raported 
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in the Appendix. 

In general, reilability astimatea ^ere quite hlghi and results showed 
that the extent of consiitancy among scoras projected by eKperts ^^as easily 
within acceptable limits. In short, projected scores w-ere reliable and rela- 
tively free of measurement error (estimated by MOVA) . Table 1 reports re-- 
liability estimates for the Bayley and KIDS la^entory. 

Table 1 

Estimated Reliability for Pxoj&ctad Scoras 



Test 

Bayley Mental 

Bayley Motor 

KIDS Cognitive/ Language 

KIDS Gross Motor 

KIDS Fine Motor 

KIDS Self Care 

Average 



Reliability Coefficient 



One E:Kpert Averag e of Four E^cperts 

,51 .81 

•84 .95 

*73 .91 

*69 -90 

.77 .93 

.10 .87^ 

*71 .90 



Coefficient based on three rather than four e^cperts dua to incomplece daCa 
froia one expert. 

Inspection, of Table 1 shows that reliabilities for any one expert ranged 
froia *S1 to ,84, and they ranged from * 81 to .95 for the average of four 
parts. The overall average reliability coefficteat for projected scores ffrom 
one expert was .71 and was ,90 for the average of four eKperts. The above 




results demonstrate the feasibility of the theoretical control strategy in 
Chat projected scores were internally CDtisistent among experts, and the 
prDporcion of measurement error in projected scores was well within acceptable 
limits* 

1.2 What was the validity of projected scores? 

The best measure of validity of course would be actual measurement 
of children taken 12 months from date of pretest without project interventioti. 
Unfortunately, it was impossible to collect such meaeurementSp Another mea- 
sure of validity would be to compare pTDjections of the current panel with 
an additional panel of experts, but financial resources did not permit such 
a comparison at the time of this study. 

Ac the present time, the major source of validity rests upon the cre- 
dentials of selected expert panel members. Inspection of the list of panel 
members on page five speaks to the high level of expertise and professional 
atacui among experts selected for the study, Ifhile the validity of ths 
Bayley Scales has been well established and while criterion validity of the 
KIDS Inventory has also been established, the validity of scores projected 
on the Bayley and Inventory rests primarily upon professional credential and 
eKperience embodied in the eKpart panel. No descripclve data about recent 
eKperience with the Bayley or with young handicapped children were collected 
from panel members* 

1.3 Did individual experts exhibit any coTisisteiit deviation in projecting 
scores? 

The above question essentially asks if one or more experts tended to 
be consistently higher or lower than other experts in projecting scores, 4 
ripaated--measures ANOVA was used Co test for a significant eKpert main effect 
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in projected scores (this mB the same ANOVA discussed under question twa). 
Results o£ the analysis revealed no sigtiif leant eKpert effects , with the 
possible eKceptlQti of scores from che KIDS self-care scale w^here the F*-ratlo 
was significant at the 10 percent level, but only a five percerit or amaller 
level la generally considered significance 

Table 2 presents the means and standard deviations of projected acores 
as well as the F^'fatio for testing significance of differtfict among avarage 
scores by eKpert s. The ANOVA suramsry tables are included la the A.ppendiK* 

The results alearly indicate that individual experts did not have any 
significant bias In projecting scores. In other words, there vas no signifi- 
cant tendency for any given aKpert to project consistently higher or lower 
than other aKperts. The data In Table 2 also show that variability in pro- 
jected scores within each expert was cotuparable across all four axpertg. An 

F test for hDmogenelcy of variance iti projected scores Cox each scale 
max o - J 

verified the above observation, 

1.4 Whs the variability in projected scores related to children's pretast 
perf ormancey related subtest performance | age, nuinbef off itetns t^tBdy 
handicap, or eKtent of available information? 

As expected J there was variability in projected scotes for any one child, 
and Table 3 reports the standard deviations among projected scores for each 
child for both the Bayley and KIDS Invetitory, These data CTable 3) indicate 
that the g^ceatast eKtent of projected score variability Vas in the Bayley 
mental scalSj and the least variability MS in the KIDS seLf-^care scale. 

While the incerrater reliability estimates (see questioti 1^1) showed 
that the degree cf varlsbiUcy was within tolerable limits^ questions of 
Interest pertained to idencifying any variables or factors which might have 
been associated ^ith score variabliicy ^mong eKparta, For eKainple s were 
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Table 2 





Means , 


Standard 
for P] 


Deviations, and F- 

cojettad Scotfes 


-ratios 






Test 




(1) 


Expert 

(2) 


(3) 


(4) 


F 


P 




M 

SD 


34.2 


100.8 

31.9 


100.5 
33.1 


96,4 
30.6 


0.14 


NS 


Bayiey mo cox 


M 

SD 


14.4 


41.9 

la.l 


40.9 
13.4 


38.8 
15.3 


i.oo 


NS 


KIDS CDgnitive/ 
language 


SD 


7.6 


34.7 
11.0 


35.6 
13.1 


31 ,8 
12.2 


J..45 


NS 


KIDS gross motor 


H 
SD 


34.1 
10.3 


31.1 
14.2 


29.9 
13.6 


28,8 
15,9 


J.. 62 


NS 


KIDS fine motor 


M 

SD 


28.1 
12.5 


27 .0 
12.3 


28.2 
10.6 


26.4 
10.9 


0.40 


NS 


KIDS self care 


M 

SD 


16.9 
6.2 




20.5 
10.5 


18.8 
8.4 


Z.94 


<.10 


^ ^ mean or aver age 


score, 


SD « atandaxd daviation, NS 


a not significant. 





^Incomplate data from expert tiiimber tvo. 
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table 3 

Standard Deviatloni Mong Projected 
Scoces for Each Child 



Bavlev lOTentory 



Child 


Mental 


Motor 


Gross Motor 


Fln,a Mo Cor 


Lanauase 


Self-Oa: 


1 




4.57 


10 ,98 


4 * 08 


0 .lo 


2 , Jo 


2 


5,yo 


5,89 


2 .52 


3p79 


o ,9o 




3 


3.11 


2 .36 


7 ,51 


4*79 


1 * 7 J 


Li ^3 


4 


14. 48 


6.61 


7 .55 


8.66 


9 ,07 


3 .51 


s 


4,51 


8>42 


6 , 70 


2 . 99 


2.43 


4, 


6 


7. 97 


1,15 


2 .94 


.2 .94 


5.94 


8.89 


7 


7,63 


1 ,50 


2.65 


3.32 


3.59 


3.30 


8 


3.20 


3.77 


2.06 


1.91 


5.00 


4.04 


9 


9.15 


3.10 


5.70 


1.53 


5.20 


2.08 


10 


7.63 


9.57 


9.68 


10.37 


4.57 


4,03 


11 


11,92 


10.60 


11.18 


9.67 


7.59 


3.87 


12 


6.40 


3.00 


2.22 


4.99 


2.99 


4.55 


13 


7,93 


1.63 


6.63 


3.00 


0.58 


4.93 


14 


7.68 


4.19 


8.54 


7.05 


6,95 


9.14 


15 


7.50 


9.57 


8,54 


3.92 


3.40 


2.65 


16 


9.33 


3.00 


2.83 


10.99 


3.70 


3.59 


17 


8.04 


4.24 


5.03 


1.73 


5.25 


5.26 



Average 7,46 4.89 6.13 5,04 4.87 4.14 

SD 2.89 3.04 3.14 3.11 2.38 2.12 
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eKperts more consistent in projecting scores for older or younger children? 
Did the type of handicap involvemenc affect expert coniiscency in projecting 
scores? 

The basic research procedure for addressitig the above questions con- 
sisted of computing correlation coefficients between the standard deviation 
of projected scores and selected vairiables* For example, a correlation 
was computed between the standard deviations raiong projected scores for each 
child (aee Table 3) and variables of Interest which were number of test 
items projected (i.e., length of test)? pretest performmnce level? and re- 
lated iubteat performance, A variacion of this procedure was used for handi- 
cap and eKtent of information reported in the child's case profile. 

Data presented in Table 4 show that chronological age was generally 
not significantly related to variability in projected scores, but there was 
a significant (p < *05) correlation with the Bayley mental scale and the 
KIDS gross motor scale. On these scales, the negative correlation cotffi^ 
cient indicated that experts tended to be less variable in their scor^ pro- 
jections with older children. While some of the remaining observed correla"* 
tions were substantially different from zero, the sample siEe CN^17) was too 
small for statistical significance. Of course, larger samples may have 
yielded smaller coefficients. 

Another interesting finding in Table 4 deals with the nuinber of items 
projected. It was thought that experts might become more variable in their 
projections as they projected more and more test itemSj the data did not 
support this hypothesis in the Bayley scales, but there was sortie support for 
the hypothesis in the KIDS Inventory scales. There was a aigiiificant correla- 
tion for the KIDS self-care scale (r ^ •58), and the correlation (r - -,47) 
for the KIDS fine motor scale was almosc ilgnificant, The pattern of observed 



Correlation Coefficients 
for Selected Ranked Variables 





topert Variability in Projected Scorta^ 


Correlated 
Variable 


Bayley 
Mental to tor 


Gross 
Motor 


KIDS Inventory 
Fine Cognlcive 
Motor Language 


Self- 
Care 


ChrQUo logical age 




-*51^ 


.29 -.14 


.20 


Number of iteM 
projected 


-.37 -ao 




-.47 -.16 


.58* 


Pretest performance 
(raw score) 




(no 


pretest data available) 



^Significant at the .05 level (p < .05* r - .48). 

defined as the total nuiaber of itenis prDjected from the basal or entry 
level for a given child to the average number of the ceiling level items 
projected, 

^Number of items wtthin each subtest ^te as follows: Bayley mental 163 t 
Bayley motor - 81, KIDS gross motor - 77, KIDS fine motor - 70 ^ KIDS 
cogtiicive/ language - 112 j KIDS self^care .64. 

correlations betweea expert variablLlty and nuinber of items projected may 
refltct the psychometric status of the KIDS Inventory as compared to the 
Bayley. Hence the tiumber of items projected tended to be associated with 
expert variability In the KIDS InveatoTry, but there was no such association 
evident in Che more sophisticated Bayley scales. 

The data In Table 4 also indicate there was a significant correlation 
between eKpert variability and pretest perforTnance level on the Bayley 
mental scale (r ^ ,54). The postive coefficient indicated that eKperts be- 
came more variable as children attalTiad higher mental developm^snt . However, 
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the trend, though tionsignif leant ^ was reversed in the Bayley motor scale 
(r ^ -•40)* Unfortunately, pretest KIDS Inventory scores were unavail-- 
able for similar study. 

Ajiother question of Interest was whether or not there were signifl^ 
cant ralatloiiships among SKpert variability in projected scores and sub'- 
scales within the Bayley and KIDS Inventory* In other words, if there 
tends to be high expert variability for given children in the Bayley mental 
scalSj does there also tend to be high eKpart variability for given chll-^ 
dren in the Bayley inotor scale? Correlational analysis showed there were no 
sueh relationships in projected scores. The correlation between Bay lay 
mental and mocor scales in terms of expert variability was .23, The cor r el- 
lations among KIDS Inventory scales (Table 5) also revealed nonsignificant 
correlation coefficients, which ranged from --,12 to .35, 

Table 5 

Correlations Mong KIDS Inventory Scales 
in Terms of Variability in Projected Scores 





Gross Fine 
Motor Motor 


Cognitive/ 
Language 


Self- 
Care 


Gross Motor 


1.00 .35 


.13 


-,12 


Fine Motor 


1*00 


.26 


.02 


Cdgnltlve/ language 




1,00 


,10 


Self-Care 






l.OO 


^Similar correlations computed between the 
was .23. (For p < .05, r ^ .48) 


Bayley mental and motor 


scores 
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The final variables of interest relative to expert variability were 
handicap and eKtent of information reportad in the child case profile. 
Since these variables were a little difficult to quantify ^ the analysis 
adopted was a Variation of correlational analysis* The basic proceduire 
was to select those children for whom experts were conslscently variable 
and leas variable in their projected scores* Scandard deviations of pro- 
jected scores (as reported in Table 3) were plotted in a frequency dis- 
tribution * and then one or more children were selected from each end of 
the distribution who were termed oucllers in each scale, Oucliers were 
decermined by visual inspection 5 and the outcome of this procedure are 
reported in Table 6, 



Table 6 
Results of Outlier Analysis 



Scale 


EKpert Variability 






Low 






High 


Bayley Mental 


1. 


2S 3% 5. 


8* 




4^ 11^ 


Bayley Hotor 




6, 7, 13 






5. 10^, 11^, 15 


KIDS gross motor 


2^ 


, 6, 7, 8^, 


12, 


16 


1, 10^, 11^ 


KIDS fine motor 




8^, 9, 17 






4^, 10-, u\ U^, 16 


KIDS cognitive/lang . 




5, 13 






2, 4\ 11^ 14^ 


KIDS self-care 


1* 


2^ 3% 9, 


15 




6, 14^ 



Children number 2, 3 5 and 8 were selected as having consistently high 
ejcpert agreement (1*6*5 low expert variability)* 

Children number 4 3 10, 11, and 14 were selected as having consistently 
low sKpert agreement (i»e*, high expert variability)* 
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Table 6 shows that children numbered 2, 3, and 8 were in the outlier 
group for low expert variability fsr at least three of the seven scales , 
and children numbered 4, 10, 11, and 14 were in the high sKpert variability 
group for at least three of the seven scales. In other words, experts 
consistently agreed' in their projections for children numbered 2, 3^ and 8^ 
and they consistenCly disagreed in their projections for children numbered 
4j 10, 11, and 14, Thus, the above procediire, yielded two small groups of 
children for further study in terms of handicap and extent of information 
reported In the case profile. 

Review of the case profiles of the above two groups of children re^ 
vealed no differences of discernible consequence between the groups. Ap-- 
parently, expert variability was not related to handicap involvement nor 
eKtent of descripcive information, but this conclusion may be misleading 
since there was relatively little variation among sampled case profiles in 
terms of Information available, and almost all handicapping conditions 
were in the severely handicapped range. 

The data in response to question 1.4 about variables associated with 
expert variabillcy in projecting scores suggested that there may be some 
types of children for whom experts are likely to agree more in projected 
scores. Results from the current sample of 17 children and 4 experts indi- 
cate that chronological age and developmental level are the variables most 
likely to influence expert variability. Psychometric sophistication of the 
instrument used for projecting scores is also throught to have some associa- 
tion with variability of projected scores. While it seems reasonable that 
some children would present a more difficult score projection taak^ the 
results of the study provided only first level investigation into this 
issue, 24 



2*1 Was th^ra significant improvement in actual pre-posC scores of 
gampled ehildren? 

While there were projected post test scores fox 17 Ptoject KIDS chil^ 
dfen, there were actual pre--post Bayley meiital and motor scores fot only 
12 and 11 child ran ^ due to atcricion in the sample ovmt the LZ^monCli in^ 
terval. As noted previously, no actual pre-post KIDS la^entory scarea 
t^era available, 

A repeated-measures MQVA was used to test Che significance ofi pre- 
post Improvemant on the Bayley mental and motor scales. Table 7 giv^a the 
results of the ANOVAs^ and Table 8 presents means and standard davlations 
for actual test scores * 

Table 7 

Repeaced^Measuras ANOVAs for 
Actual Pre-Post Bayley Raw Scoces 



Scale 



Source 



df 



MS 



Hsntal 



Mo tor 



Pra=pDSt 
Residual 

Pra-post 
Residual 



1 
11 

1 
10 



19795.04 
163,04 

2978,91 
72,51 



66.21 



41,08 



< ,001 



^ ,001 



Inspection of Tables 7 and 8 clearly shows that sampled childrin made 
higlily signlf leant pre-^post imprDVanient , ConversiDn of average pre-post raw 
scores to developTnentai ages indicated an average gain of about elghc and 
ona-half months on the metital scale and about severi months on Che nLOCor 
scale* The above results do not necessarily indicate that the observed jLni'- 
provament was due to project inCQTvention, but chef do shov that sampled ehil- 
dren gained while In the project. 
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<0 



Table 8 



Means and Standard Deviations 
for Actual Pre--Post Bayley Raw Scores 



Mental 



^totor 



pre 



po 5 1 



pre 



Mean 



70.58 



113.00 



24.18 



47,45 



Standard deviation 



36.27 



42,46 



16,59 



21.21 



2.2 What ware the characteristics of theoretical control scores? 
Questions 1,1 through 1*4 of this report spoke to psychometric vigor 

of scores projected by the panel of experts , and these data supported the 
feasibility of adopting a projected score as a control score for comparison 
to actual posttest scores. Since pre-ppst KIDS Inventory scores were unavail 
able^ the theoretical control strategy necessarily relied on projected Bay^ 
ley scores. 

Reliability analyses (see question 1.1) showed that the reliability of 
the average Bayley score projected by four experts was a ,81 for the mental 
scale and ,95 for the motor scale. This appeared Co justify adoption of 
the average projected score for each child as the theoretical control score. 
On the mental scale, control scores ranged from 43,7 to 140,0, and the mean 
and standard deviation were 104.1 and 33,1. On the motor scale, control 
scores ranged from 31.5 to 141,3, and the mean and standard deviation were 
40,7 and 14,6. 

2.3 How did theoretical control scores compare to actual posttest scores? 

A repeated-^measures MOVA was adopted as the statistical model of choice 
to test the hypothesis of no significant difference between actual posttest 
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Bayley performance and theoretical control performance on the Bayley, This 
model might at first appear inappropriatep since the design of the theoret^ 
ical control strategy can easily seem to resemble the traditional indepen- 
dent groups model consisting of two groups, one control and one eKperimental. 
In such a design, one would employ the one-'way ANOVA for independent groups* 
However, the repeated-^measures model is clearly the appropriate statistical 
model, since the actual posttest score and the theoretical control score 
are dependent upon the same observational unit i±>B>^ the child). 

The results of the repeated-measures ANOVA yielded nonsignificant 
differences (p < .05) between the control and actual posttest Bayley scores* 
Howavar, the observed difference for the mator scale approached significance 
at the .05 level* Table 9 gives the ANOVA summaries and Table 10 presents 
relevant means and standard deviations. 

The data in Tables 9 and 10 show that the trend in observed differences 
supports a hypothesis favoring project intervention in that the actual scores 
were higher than the control scores. While the observed differences were not 
significant at the *05 level, motor scale improvement was significant at the 
.10 level* Hence, one would reject the null hypothesis of no treatment ef- 
fect for motor scores at the 90 percent confidence level, but the 95 percent 
level is traditionally the lowest acceptable confidence level. 

The data in Table 10 show that the observed difference between control 
and actual average scores on the motor scale was 6.19 units. Even though 
this difference may appear to be rather large ^ one typically interprets the 
magnitude of differences relative to group variability. The best way to do 
this is to form a ratio comparing the observed difference to the sample Stan-* 
dard deviation. In the case of motor scores, this ratio equals ,37. 
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Table 9 



Repeated'-Measures MOVA Comparing 
Theoretical Control and Actual Posttest 
Bayley Raw Scores 



b 

Scale Source _df MS F 



Mental 


Contro 1-actual 


I 


429.26 


2.78 


NS 




Residual 


11 


154.27 






Motor 


Control-actual 


I 


210.80 


3.87 


<.10 




Residual 


10 


54.44 







^df = 1, 11, F « 4.84, p < .05; df « 1, 10, F * 4.96, p < .05; df « 1, 11, 
F - 3.23, p = < .10; df = 1, 10, F « 3.28, p < .10. 

There were actual poattast scores available for 12 children on the mental 
scale and for 11 on the motor scale* 



Table 10 

Means and Standard Deviations 
for Theoretical Control and Actual 
Posttest Bayley Raw Scores 



Mental Motor 

Theoretical Theoretical 

^ Control Actual Control Actual 

Mean 104.54 113.00 41,26 47,45 

Standard Deviaiton 33,20 42,46 15,19 21,21 
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(The sample standard deviation was obtained by averaging variances for the 
control and actual scores,) 

The first reaction to the above ratio of *37 may be chat the observed 
difference was not even of practical significance, let alone statisGical 
significance, since a difference of one^third a standard deviation or less 
is usually considered trivial. However, consideration should be given to 
the special nature of the subject population in the study. In the usual 
acad^^mic settings one==thlrd standard deviation's difference would tnean, X 
number of additional test items correct. These items would likely pertain 
to such things as the correct answer to 'how much does 4 k 12 equal' or 

m 

'what is the largest continent' or 'what is the correct spelling of cat?' 
In the case of Project KIDS^ Bayley test items dealt with such things as 
responding to a verbal request or standing unassisted. Thus, one can see 
that the usual method of ascertaining practical significance may not apply 
when test items call for behaviors of greater practical significance* 

As noted earlier the average Bayley control score on the motor scale 
was 41, and the average actual score was 48 (rounded upward). The behaviors 
contained in the additional seven motor items arei walking with help, sit- 
tings playing pat^a^cake, standing alonc^, walking alone, standing up, and 
throwing a ball* 

The Implication of the foregoing discussion Is that getting an item 
such as 'standing up' correct has a great deal more practical significance 
than an item such as 'how much is 4 x 12?' Hence^ the practical signifi- 
cance of the observed difference in motor scores between the theoretical 
control and actual scores is probably much greater than the computed ratio 
of .37 Indicates, (Of course, the additional behaviors mastered by children 
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would vary considerably, The above behaviors contained in the consecutive 
items between control aiid actual score averages were presented only as 
examples). Porter, Schmidt, Floden, and Freeman (1978) have also recently 
called for the interpretation o£ effect size in terms of substantive pro- 
gram goals and substantive characteristics of the criterign test. 

The above line of thought leads one to consider a 90 percent confi- 
dence level for rejection of the null hypothesis of no treatment effect 
in the motor domain. This researcher recommends adoption of a 90 percent 
confidence level (or .10 significance level) in this particular situation 
because of the extremely small sample size (N-11, motor) and because of 
the practical significance of differences between means which are small in 
a statistical sense* As is well knovm, the F^test can be very powerless 
in the case of small sample sIes and small differences between means and 
thereby unlikely to detect differences which would be statistically signifi- 
cant with a larger sample or difference. One simple way to increase statis- 
tical power is to reduce the confidence level,, and such a procedure appears 
warranted in this situation. 

The analyses in response to question 2,3 indicate a significant 
(p < .10) effect favoring Project KIDS intervention in terms of improved 
progress in motor development. In other words, the actual posttest motor 
scores of project children were significantly greater than predicted by the 
panel of experts. Hence, one can conclude that observed progress in the 
motor domain was greater than would have been eKpected from normal growth 
and maturation* 

While similar logic might be applied to the observed gains in mental 
scores, it was inappropriate to do so, since differences between mental actual 
and control means were not statistically significant (p < ,10). These 
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results might appear confusing in that one might expect intervention to be 
mora effective in the mental domain than in the motor domain. At this time 
there is no clear explanation available except that expert panel members may 
have had a similar expectation and thereby projected more conservatively In 
the motor domain. Of course the basic assumption of the study was that ex^ 
prectations Ci,e,, projections) of panel members were valid* 

One interesting issue raised by the foregoing discussion is i,rhether 
or not the same rationale should be applied to the test for an expert main 
effect (see question 1.3* page 14) and the answer would seem to be yes* 
Inspection of Table 2 (page 16) shows that the ratios of the range of mean 
for experts to group variability were generally quite small for all sub^ 
test except the KIDS cognitive/language and the KIDS self-care scales (34 
percent and 42 percent respectivaly) , The F^ratio does not approach 
significance for the cognitiva/lauguage scale, but it is significant at the 
#10 level for the self=care scale^ Visual inspection of expert means shows 
that experts three and four were comparable but that expert one differed 
substantially from expert three. 



The following lists the conclusions and results of the study i 
1. Internal consistency reliabilities of projected scores for both 
the Bay ley and KIDS Inventory ranged from •81 to .95, and these reliabil-- 
itles were considered to be very good, especially in light of the develop- 
mental nature of the theoretical control strategy. The following gives 
the reliability coefficients for each test scales 



Conclusions 
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Bayley mental .81 

Bayley motor .95 

KIDS CO gtiitive/ language ,91 

KIDS gross motor .90 

KIDS fine motor .93 

KIDS self-care .87 

2. The study did not empirically investigate the validity of scores 
projected by the expert panel. However, the credentials of experts on the 
panel were judied to have been sufficient to warrant a fair degree of con- 
fidence in projected score validity, AdditionaUy, the reported content 
validity of both the Bayley and KIDS Inventory were sufficiently highj and 
this factor would logically lead to increased validity in projected scores. 

3. There were no significant differences among experts in projected 
scores except for the Bayley self-care scale (p < .10). On all other scales, 
any given expert was not significantly higher or lower than another in his 

or her score projections, (For the rationale behind adoption of the 10 
percent significance level, see conclusion number six.) 

4. A.S expected, there was variability in acoras projected by experts 
for any given child, and the extent of projected score variability differed 
across children. In other words, experts varied more in projecting scores 
for child A than for child B, for example. i'Jhile it seems reasonable that 
some children would present a more difficult score projection task, the 
results provided no definitive answer as to which factors might Influence 
expert variability. Chronological age and developmental level were seen 

to have some association with expert variability, but handicap, number of 
test items projected, and extent of available inforniatiQn were relatively 
independent of expert variability. 

5. Sampled children made highly significant Improvement in develop- 
mental progress during the la-month pre-post observation period. On the 
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average s children gained eight and one-half months on the Bay ley mental 
scale and seven months on the Bay ley motor scale, 

6, Comparison of cheoretical control scores to actual Bay ley scores 
showed that the Project KIDS children performed significantly (p < ,10) 
better than the control in the motor domains thereby indicating that motor 
gains ware greater than would have been expected without project interven- 
tion* The reader should not discount this significant findings even though 
the significance level is not at the usual ,05 or *01 level, Rathers pro- 
ject intervention should be considered to have made a significant difference 
just as if the significance level had been ,05 or .01. (The rationale for 
the above conclusion is based on two statistical concepts known as 'power' 
and 'practical significance** Power has to do with the probability of find- 
ing a significant difference when in fact project Intervention is truly effec- 
tive* If the power of a statistical test is low, there will not be much 
chance of finding a significant difference between groups even when the treat- 
ment is effective* Practical significance applies to the practical meaning 
of any observed significant difference. In some cases, power can be so great 
that a very small difference of no practical significance can be statistically 
significant* ^ important point Is that small differences require considerable 
power to be statistically significant* In the case of the Project KIDS evalu- 
ation, the difference between the control and treatment scores was relatively 
small, about one--third a standard deviation^ which would usually be considered 
to have no practical significance* However, Project KIDS served a unique 
"*^population, unlike that encountered In the usual public school evaluations 
of reading and math programs* (^ile test items measuring impact of a lan- 
guage arts program would test skills in word recognition, spelling, and so 
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forth, test items measuring impact of Project KIDS tested skills in such 
basic behaviors as iitcing and standing. Thus , improvemenC in even a few 
teat Itecas would seem to have much more practical significance than in the 
case of the usual language arts program. Thus, the statistical test in the 
case of Project KIDS needed to be relatively powerful in order to detect 
even a small difference between the control and treatment groups. The 90 
percent confidence level was adopted Co increase power and to detect any 
observed difference which would have been significant given a larger sample 
size in the project evaluation.) 
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PROJECT KIDS 
'nmomtlCAL CONTROL GROUP STUDY 



DIRECTIONS TO EJ^ERTS 



The following details the content of your materiale packet and 
cribea procedures for making and recording your projections * 

PROJECT KIDS EVALUATION DESIGN 

Thii document ia included only for your inforaation about 
the project evaluation in general and more specifically 
about the design of the theoretical control group at\3dy 
(see pp* 17-25 for diacusaion of the theoretical control 
strategy) . 

INDIVIDUAL CASE REPORTS (N^l?) 

There is one case report for each of 17 children^ Each case 
report contains the following information: 



1, Subject's Suimary Sheet (one page) 
2* Project KIDS Referral Fora (one page) 

3. Fmily InfoMatlon Form and Home Assessment 
(usually 10 pages) 

4. Danver Developmental Screening Teat (one page) 

5. Bayley Scales of Infant Development (variable pages) 
(These give the Bayley pretest results*) 

6. Medical Data (variable pages) 

7* University Affiliated Facility Sunmiary 
(variable pages) 



BAYLEY MCOra FORMS 

This includes 17 copies of the Mental Scale Record form m.d the 
Motor Scale Record Fom for the Bayley Scales of Infant DevelopMnt* 

Directioha for recording your projections on the Bayley Record 
(both Mental and Motor) forass 
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1, Note the PROJICTED CA on the Subject's Sijmmary Sheet. 
This ie the age th© child would be at twelve months 
past the ttoe of pretaatlng with the Bayley (pretest 
results for each child are included Ija the individual 
case report - Item B,5.) 

2* Based on the total relevant infoOTation contained in 
each child's individual case reports project what you 
think each child's performance will be on the Bayley 
Mental and Motor Sc^es at time of the PROJECTED CA. 
hmBimm that there has been no special Cr^edlali 
therapeutic, educational i clinical, etc) Interventions 
into the child's life* In other wordsi how might the 
child have progressed had he lived in hia family during 
the twelve month period without benefit of services 
other thto those his or her family could provide? 
Please assime that any medical Intervention necessary for 
life support which was present at the start of the 
twelve month period was conttoued or that any life 
support Intervention which might become necessary during 
the twelve months was made available to the child, 

3* Record your projections (i.e., check the items "P" or 

"F*') on the Bayley Record Forms for each child as though 
you had actually atoiniatered the Bayley to each child i 
Use the pretest ceiling level as the basal level for 
the starting point in ^'administering-' your projected 
Bayley* 

temple: if a child's ceiling level on the pretest 
Mental Seals is item 47, then the basal level for begin- 
ning the projected Mental Scale would be item 47. Pleaae 
note that the basal item for the projected Bayley would 
be the last item gassed prior to termination of the pre^ 
test Bayley, 

Please follow the Bayley Manual specifications for 
determining the ceiling level of your projected Bayley 
(i.e. s a criterion of 10 successive items failed on the 
Mental Scale and 6 successive items failed on the Motor 
Scale; Manual for the Bayley Scales of Infant Development ^ 
the Psychological Corporation, 1969 p p. 29)* 

It is not necessary for you to compute a composite 

score (either raw or normed conversion) as we will do that 

here in our office. 

D, KIDS INVENTORY OF DEVELOPMNT 

This includea one copy of the KIDS Inventory of Development manual 
and a copy of the KIDS Inventory of Development record fora for 
each child (N^17) • 
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KIDS Inventory of Development - MMUAL 



This doaisment descrlbei the development and rationale 
behind the Inventory , as well aa giving a description 
Mid direetiona for administering individual items in 
the Jtoventory. 

KIDS Inventory of Development - RECORD FORM (N^17) 

The RECOM FOEM eontaina four major devBlopmental areas; 
gross motor (Ql)p fine motor (M) , cognitive/language (CL), 
and self^oare CSC)# Within each area, Inventory Items 
are grouped according to ahronologioal age intervals, 

Dlreatlons for reeording your projections on the KIDS 
Inventory of Development* 

a. Use the same PROJECTED CA (see Sub j act "S 
Smmary Sheet) that you uaed for each child in 
making the Bay ley projeetionsi 

b. Project what you think eaeh child's performance 
on the Inventor would be at the time of the 
PROJECTED CA (which is 12 months past the age of 
Bayley pretesting) * Use the saffle rationale and 
assumptions used in making the Bayley projections 
(see item C,2.) * 

c* Record your projections for each child on m 

Inventory RECORD FORM as though you had actually 
administared the Inventory to each child, (Note 
that your materials packet does not contain any 
pretest Inventory rasultaf) 

When "adminiatering" the Inventory, testing should 
be initiated at the level at which the child is 
expected to achieve. A basal is established when 
a report of P (pass) is recorded for all items within 
one age interval before the first failure . Testing 
should continue unt^l the child has failed all items 
within one age interval, Indicating a ceiling # 

Observe that projections must be made within each of 
the four developmental areas (GM, FM, CL, md SC) of 
the Inventory* There is no need for you to figure 
a cdmposlte Inventory score for the child* 
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* * * * RETUmi 01 MATERmS 5% * * * 



TDffii 



At your aarllaat coavenience. 



PLACE 



Return materiale to 



Dr« Ruth Turner 

Aisist^t Dlrector-Speeial Education 
Dallas ISD 
3700 Roia Ava* 
Dallas, Taxai 75204 



WHATj It is only necessary to raturn the record forms for each ahlld. 



recQrd form for each child . Fleaae group the record foras hy 
child according to this order within each child ^ Bayley tontal, 
Bay lay Motor, KIDS Inventory * (Please include some Identifying 
information in your return correipondancei so that we know 
for sure which record forms are from which expert*) 

The individual case reports may be retained if desired (as well 
as the KIDS Evaluation Plan and KIDS Inventory) * You may wish 
to refer back to the case reports when the results of the study 
become available* 

If you have any questions, please feel free to call Dr. Daniel Macy, 
Department of Research ^ Evaluation, Dallas ISD (phone 214 - 321^2667) » 
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ANOVA SUMMARIES 





Qrt t% 1^ r* B 


df 


MS 


F 




V 


Bayley Mental 


Between subjects 


16 


2804.50 


6.25 


< 


.001 




Within sub j ects 


J J. 












Between experts 


3 


67,39 


0.14 




NS 




Residual 


48 


472.63 








Baylay Motor 


Between subj ects 


16 


750.51 


22.99 


< 


.001 




Within subjects 




iL , 










Between experts 


3 


32.57 


1.00 




NS 




Residual 


48 


32.65 








KIDS cognitive/ 


Between subjects 


16 


403.53 


12.22 


< 


.001 


language 


Wltnin subjects 


j1 










Between experts 


3 


46.76 


1.45 




NS 




Residual 


48 


32.16 








KIDS gross 


Between subjects 


16 


581.36 


10.21 


< 


.001 


motor 


Within subj ects 


51 


Do * ^ J 










Between experts 


3 


89.08 


1.62 




NS 




Residual 


48 


54.92 








KIDS fine 


Between subjects 


16 


445.75 


14.83 


< 


.001 


motor 


Within subjects 


51 


30.05 










Between experts 


3 


12.31 


0.40 




NS 




Residual 


48 


31.16 








KIDS self- 


Between subj ects 


16 


180.52 


8.42 


< 


.001 


care 


Within subjects 


34 


21.45 










Between experts 


2 


56.61 


2.94 


< 


,10 




Residual 


32 


19.25 
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Reliability Computations for Expert Panel 
(See question 1.1) 

The following presents the reliability computationa for determining 
the Intraclass correlation coefficient derived from the analysis of 
variance model. The basic procedure was to compute an unbiased estljnate of 
theta (0), the term used to denote the ratio of true score variance to 
error score variance, where 

MS^ ^ (X) (MS^) 

0 - -- _ ^_ where 

k (X) (MS ) 
w 

k is the number of ^pertSj 

MS, is the mean square between subjects 

from the MOVA^ and 
MS. is the mean square within subjects 
^ from the MOVA, and 

X = ^ ( k^l) where n is the 

n (k-1) -2 

number of subjects. 

The above term, 0, was then used to compute the intraclass correlation 
coefficient as follows: 

0 

"1 1 > 0 ' 

The above equation gives the reliability estimate for a single expert. The 

Spearman-Brown prediction formula was used to obtain the reliability estimate 

for the average of all four experts » 

r - 4 0 , 
4 1 + 40 

For a more detailed descripcion of the above procedure , see Winer (1971 , 
pp, 283ff-)* The ANOVA model also permits adjusting the error variance to 
partition out that variance attributable to "expert" main effects. This was 
not done since there were no significant eKpert main effects in any of the 
test scores (refer to MOVA aumiarias on previous page). 
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